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ABSTRACT 

This issue contains five articles about assessment in 
adult literacy programs. The first article, ’’Living Literacy: 
Rethinking Adult Learner Assessment" (Susan L. Lytle), focuses on 
developing alternative forms of assessment such as literacy 
portfolios to identify changes in four dimensions of adult literacy 
over the course of literacy programs: learners' beliefs, practices, 
processes/products, and goals/plans. In "CALPEP: The California Adult 
Learner Progress Evaluation Process" (A1 Bennett), the objectives, 
key components, and use of CALPEP are outlined and a sample page is 
presented. Project REACH, a basic skills enhancement program for New 
York state employees, is profiled in "Assessing Workplace Literacy: 
Asking New Questions" (Carol D. Young). Discussed in the article "An 
Assessment Overview" (Kate Toms) are the problems involved in gaining 
acceptance of alternative assessment practices such as portfolio 
assessment and the comparative validity of alternative and 
standardized assessment. The benefits and potential drawbacks of 
using portfolio assessment in adult literacy programs are considered 
in "Portfolio Assessment" (Barbara Van Horn). Also included is a 
brief description of available portfolio assessment software. (MN) 



************************* ****** ****** ********************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

****************** ******** ********************************************* 



r~~ 

r— l 

m 

in 

o 



8 



WM,tR,AL has been. granted by 



■ jf AKltfMf irm ^= 

Emm jU iN°FORMf?.o cATioNALREs ° uRc| 

Practitioner 

A Publication of Literacy Volunteers of America — New York State, Inc. 



Assessment 

Issue 




Vol. 2, No. 1 
August, 1994 



u s DEPARTMENT OF EDUCATION 

On<» O' i »rm I'no'owmOTt 

XjCADONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

Tlx* oocum«ot ft a* qeoo roproducM as 
^n»cefv«j from me person o* organization 
Of *0*08 ling it. 

Mmof changes haw Been made to 
improve reproduction quality 



Pomu of mew oi op«n*on» stated m trws 
document do not necetsanty represent 
official OE Rt position or policy 



Living Literacy: 

Rethinking Adult Learner Assessment 

Susan L. Lytle 
University of Pennsylvania 



Many literacy practitioners — teachers , tutors , and administrators — have become interested in new forms of assessment, in 
alternative ways of capturing the complex interactions of teaching and learning and their outcomes for adult learners. But rethinking 
learner assessment involves more than new methods; it also entails rethinking what we understand about literacy ; about learners , 
about learning, and about what counts as progress. In this article , adapted from a longer piece (Lytle, 1991 ), 1 present a framework 
teachers and tutors can use to develop approaches to assessing literacy learning in adulthood that are congruent with the ways 
literacy is lived in families, workplaces, and communities. 

As practitioners work together to learn about literacy development from and with adults there are many implications for 
revising our current curricula, programs, and policies. This framework invites discussion and debate, and particularly, conversations 
with learners and the rich documentation of the practices of teaching and learning. 
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Portraying Adults as Literacy Learners 

How we imagine literacy development in adulthood 
depends on our underlying assumptions about the characteristics, 
abilities and problems of adult learners. There is considerable 
evidence that the commonplace images of incompetence and 
marginalization, of adults who are weak and embarrassed and 
who seek the ‘privacy’ and anonymity of one-to-one tutoring, 
do not match the adults who actually come to literacy programs 
bringing with them self-concepts, interests, and literacy abilities 
as varied as those of any other group in the population. In an 
oft-cited study, Fingeret (1983) has shown how adults operate 
within complex social networks in which they are interdependent, 
offering skills of their own in exchange for the literacy skills of 
others within their network. She concluded that “illiterate” adults 
do not necessarily fit the stereotypes of dependency, weakness 
and failure affixed to them by mainstream culture. 

In accordance with the public images of helplessness and 
incompetence, many adult learners initially present themselves 
at literacy programs as unable to read and write. At the same 
time, however, they often arrive with prior experiences with 
authentic reading and writing as well as stories recounting 
extraordinary endeavors to teach themselves strategies for using 
written language in their daily lives. Often these efforts have 
been collaborative, utilizing a network of family or friends and 
requiring that these learners experiment, take risks, and assume 
increasing responsibility for their own education as writers and 
readers. 

To understand literacy development in adulthood, we also 
ted to look beyond the experiences of adults enrolled in 



programs to explore the functions and uses of literacy by adults 
in a range of contexts including families, workplaces, churches, 
and community organizations. Assuming competency rather than 
deficiency means looking outside of programs to identify 
instances of adults with limited abilities in reading and writing 
functioning in complex social environments requiring a range 
of intellectual and social skills. 

The purposeful use and development of literacy needs to 
be seen within the broad contours of a person’s personal and 
social world. And although much of the theoretical literature 
has suggested that adult learning is characterized by 
independence and self-direction (e.g., andragogy as developed 
by Knowles, 1970; 1979 and others), there is some concern that 
these studies of adults as learners have been culturally specific, 
i.e., limited to lower-middle, middle and upperclass white 
Americans (Brookfield, 1986). There may be diverse routes into 
literacy, and in different cultures or social groupings such learning 
may revolve more around joint work and interdependence than 
around individual initiative. 

What Counts as Learning — In Programs and Out 

Portraying adult literacy learners as participants in diverse 
social and cultural contexts involving a range of literacy practices 
provides an alternative to the current public image of adult 
learners as incompetent individuals needing remediation in a set 
of pre-determined technical skills. This more complex view of 
adults as learners and of literacy as practices and critical reflection 
leads next to concerns about how adult literacy learning is 
constructed in different programs. At issue here is what adult 
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learners, and the designers of literacy programs, count as learning, 
both in and out of formal instructional contexts. 

Just as in K-12 schooling nationwide, adult literacy 
programs differ from each other dramatically, and thus offer to 
the adults who attend them qualitatively different opportunities 
to learn. Programs differ in the extent to which they emphasize 
solitary or more collaborative opportunities for learning, and in 
their emphasis on teaching predetermined sets of skills or, 
alternately, in building on the literacy practices of everyday life. 
Learning in what have been identified as individually-oriented 
programs (Fingeret, 1984), for example, is organized to empower 
adults through learning literacy skills with services planned and 
provided by a staff of professionals and volunteers. Community- 
based programs (Fingeret, 1984) are more content-oriented in 
their focus on community issues and on the ability of groups 
and individuals to empower themselves through collective action. 
Some family literacy programs are geared to what Auerbach 
(1989,1990) calls the 
“transmission of 
school practices 
model, ” while others 
build on the cultural, 
linguistic and liter- 
acy practices already 
and always ongoing 
in family life. Some- 
times it is difficult to 
see immediately what counts as learning in a program. So called 
“real-world materials” may be used as texts for traditional school 
tasks, e.g., to teach technical skills in the abstract, or they may 
be grounded in the learner’s immediate experiences and provide 
occasions for dealing with issues related to using literacy in the 
world. 

Participatory educational approaches, as described by 
Fingeret and Jurmo (1989) involve learners not only in 
negotiating the curriculum but also in program management 
and governance, thereby providing opportunities for developing 
literacy within the program as an organizational, not just an 
instructional setting. As Fingeret and others have pointed out, 
adult learners bring to these opportunities prior experiences as 
participants in oral subcultures where they work interdependently, 
learning, sharing, and using information to provide mutual 
assistance. These rich patterns of social interaction often go 
unrecognized as contexts for literacy learning, as the skills of 
collaboration are unrecognized as resources for learning. 

Participatory literacy education is centered in learners’ 



“characteristics, aspirations, backgrounds and needs” and in the 
collaborative relationships among learners and program staff. 
While in traditional programs learners are conceived as recipients, 
in participatory education they “define, create and maintain” the 
program. Although relatively few programs currently can be 
characterized as fully participatory, and little research has been 
conducted on adult learning in participatory contexts, the profiles 
of literacy development of adults participating in such programs 
are likely to be radically different from profiles of development 
in traditional settings. Participatory models challenge the 
conventional wisdom about what so-called low literate adult 
learners can accomplish in their daily lives. 

Dimensions off Literacy Development 

The four dimensions of literacy we have been investigating 
include adults’ beliefs, practices, processes/products, and goals/ 
plans. Beliefs includes adults’ theories or knowledge about 
language, literacy, teaching and learning. By practices, we refer 
to the range and variation of learners’ literacy- 
related activities in their everyday lives. By 
processes we mean adult learners’ repertoires 
of ways to manage reading and writing tasks 
and the products of these transactions. Finally, 
plans signal what adults themselves indicate 
they want to learn, including their short and 
long-term goals and how they plan and interact 
to further attainment of these goals. By 
collecting data with adults while they participate in programs, 
we are describing the range and variation of each dimension and 
exploring their interconnections. This process entails looking at 
both intra-individual differences (variation in one adult over time) 
and inter-individual differences (variation across adults). 

Adult Learners’ Beliefs 

The first dimension of literacy development focuses on 
learners’ beliefs, adults’ own evolving conceptual frameworks 
or theories about language, literacy, teaching and learning. This 
aspect of literacy development foregrounds adults’ prior 
experiences, attitudes and current understandings of: their own 
social, cognitive and linguistic resources for learning and of 
themselves as gendered, raced and classed readers, writers and 
learners; the forms and functions of written and oral language; 
the nature of reading and writing processes in relation to different 
tasks, texts and contexts; and the ways that writing and reading 
are or should be acquired, learned and taught. The work so far 
suggests that adults’ beliefs may function as the core or critical 



“The work so far suggests that 
adults’ beliefs may function as the 
core or critical dimension in their 
movement toward enhanced 
literacy.” 
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CALPEP: 

The California Adult Learner Progress 
Evaluation Process 

Al Bennett 

Literacy Specialist, California State Library 
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What is CALPEP? 

CALPEP is the learner progress assessment tool used by 
the California State Library in the California Literacy Campaign 
(CLC). It is unique in the world of assessing adult literacy 
progress, and because of its uniqueness it may easily be 
misunderstood. 

• CALPEP is learner-focused, goal-oriented and user-friendly. 

• CALPEP is not a test! There is no pass, fail or reference to 
grade levels. 

• CALPEP offers tutors and learners the opportunity to review 
what they’ve done during the past six months, and to plan 
where they want to go next. 

• CALPEP provides valuable information for tutor and learner 
as well as for literacy program management at the state and 
local levels. 

• CALPEP provides uniform, generic data on learner progress 
that is independent of specific curricula that can be aggregated 
at the local and state levels. 

CALPEP provides information of value at three levels. 
First, it helps the tutor and learner understand what progress is 
being made in the learner’s literacy development. Second, it 
provides information that can be used at the local program level 
to strengthen such activities as tutor training and tutor/leamer 
support. And third, it is used at the State level to provide 
information to the State Library, the California Library Services 
Board and the legislature on the effectiveness of the California 
Literacy Campaign. 

CALPEP was designed to provide a semi-annual summary 
of learning outcomes for adult literacy learners in the public 
libraries participating in the CLC. The CLC is a public library- 
based, volunteer literacy program funded 
under the California Library Literacy 
Services Act and the Families for Liter- 
acy Act, both of which amended the 
California Library Services Act. The 
CLC offers basic literacy (as opposed to 
English as a second language) in- 
structions for out-of-school youth and 
adults. There are currently 91 participating public library 
jurisdictions throughout the state of California. 

The development of CALPEP began in 1987 when the 
California State Librarian awarded a contract to Educational 
Testing Services (ETS) to recommend a means of evaluating 
learner progress in the CLC. ETS surveyed the 46 library literacy 
ograms then in the CLC, and concluded that although each 



reported some kind of learner outcome assessment, most were 
implemented by tutors to meet the tutor’s perceived need, often 
using informal means. However, there was no single measure of 
learner progress used in local programs that ETS could 
recommend for use in all CLC programs. The lack of agreement 
on a single learner progress assessment tool was in part a result 
of the California State Library staff’s encouragement of diverse 
approaches to literacy development in the various local programs. 
Since CLC’s beginning in 1 984, staff had emphasized innovation 
and responsiveness to community needs rather than use of a 
uniform statewide curriculum. 

ETS undertook a nationwide search for an appropriate 
assessment instrument. They gathered data from 31 different 
programs in 1 8 states in a telephone survey. The national findings 
paralleled what had been found in California: there was 
dissatisfaction with standardized tests traditionally used to 
measure learner progress in adult basic education/literacy 
programs, even though many used such tests to comply with 
grant requirements. 

ETS selected two programs from the broader national 
survey for in-depth study. They were chosen because of their 
creative efforts to deal with the dissatisfaction in standardized 
testing. ETS then developed a “Discussion Paper” based on their 
findings that provided the basis for a seminar of experts selected 
because of their involvement in addressing the issues raised 
during the nationwide study. 

ETS also worked with a Field Advisory Committee of CLC 
Coordinators from eight geographical regions of California. They 
provided feedback to the various drafts of CALPEP that ETS 
developed in 1987 and 1988. 

How CALPEP Works 

Tutors and learners complete a CALPEP assessment in 
June and December. Completed forms are sent to each 

participating 
library’s office 
where the infor- 
mation is entered 
into a database: 
CLC-Base (a data- 
base management 
system developed 
by ETS as a comprehensive literacy managements tool), or a 
separate CALPEP software program now under development. 

Semi-annual CALPEP reports complement monthly 
reports tutors submit on numbers of instructional sessions, hours 
of instruction provided and anecdotal comments about learner 
progress. Tutors keep copies of CALPEP reports to use in each 
successive CALPEP assessment so activities and 



“CALPEP was designed to provide a 
semi-annual summary of learning 
outcomes for adult literacy 
learners.” 



accomplishments can be compared to plans made at the previous 
assessment. 

Each local literacy program office downloads data gathered 
semi-annually to a disk that is sent to a central location for 
statewide analysis. ETS has provided this service since 1989. 
Reports entitled “CALPEP: Analysis of Adult Learner Progress” 
are prepared for the State Librarian, and for each of the local 
literacy programs. 

CALPEP was reviewed and revised by ETS through a 
participatory process in 1993. The new version limits the scope 
of CALPEP to only the semi-annual reporting cycle, rather than 
a complete literacy program management format as in the original 
version. It has been formatted to be more user-friendly, with the 
addition of graphics in the sections assessing reading and writing 
practices. The former analytical framework of R-W-S-P-A 
(Readiness, Words, Sentences, Paragraphs and Application) was 
dropped and replaced with five options relating development of 
facility in reading and writing, and three options in progress 
toward goal accomplishment. The new version has met with 
enthusiastic reception by CLC participants. 



For further information on the use of CALPEP in 
the California Literacy Campaign, contact: 

A1 Bennett, Literacy Specialist 
California State Library 
1001 Sixth Street, Suite 300 
Sacramento CA 95814 
Tel: 916-322-0377 
Fax:916-323-1870 

For futher information on ETS’s involvement, or on 
CLC-Base, contact: 

Educational Testing Service 

2 Renaissance Square 

40 North Central Avenue, Suite 700 

Phoenix AZ 85004 

Tel: 602-252-5400 

Fax: 602-252-7499 
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In 1991 the LVA-NYS Board of Directors identified learner 
assessment as a priority area for research and enhanced affiliate 
technical assistance. This led LVA-NYS on an extensive search 
for existing alternative learner assessment instruments and 
approaches. This culminated in identifying CALPEP as the 
primary instrument designed for and being used by volunteer 
tutors and learners to collaboratively assess the student’s 
O 'ess toward reading and writing goals. It allows students to 
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report on changes in their literacy practices, on changes in their 
perceptions of their literacy progress, and ultimately on the 
impact that literacy improvement has made on their lives. 

LVA-NYS is piloting a revised edition of CALPEP as part 
of the federally mandated Indicators of Program Quality with 
the New York State Education Department. 

We gratefully acknowledge the permission of the California 
State Library to use CALPEP. 

O 



Sample page from CALPEP (adapted by LVA-NYS) 




Reading Materials 

"Here are some things that adults read, that we could read In our sessions together. Tell me If you 
read these kinds of things. If you do, how often do you read each of these per week, and how hard 
is it for you to read each one?" 

How often per week. . . How hard to read. . . 

Easy A little Very 

d o d 

Easy A little Very 

ood 

Easy A little Very 

d o d 

Easy A little Very 

d o d 



N/A O 1-2 3-4 5-6 7 

Labels/Instructions | | | | [ | [ | | | j~~ j 

0 N/A o 1-2 3-4 5-6 7 

Street names/Traffic signs | | | | [ | [ | [ | | | 

i N/A O 1-2 3-4 5-6 7 

] Bank ATMs/Pay check stubs || | | | | | | | | | | 

N/A O 1-2 3-4 5-6 _7 

□□□□□□ 



1:1 n 1:1 



Sfi* 



Mail/Bills/Letters 





N/A o 1-2 3-4 5-6 7 

Books you read to your child | [ | | | | | | J | | j 

H O 1-2 3-4 5-6 7 

□ □□□□ 

N/A O 1-2 3-4 5-6 7 

□□□□□□ 

N/A O 1-2 3-4 5-6 7 

□ □□□□□ 

N/A O 1-2 3-4 5-6 7 

□□□□□□ 

H O 1-2 3-4 5-6 7 

□ □□□□ 

N/A O 1-2 3-4 5-6 7 

□□□□□□ 

N/A O 1-2 3-4 5-6 7 

□□□□□□ 

N/A O 1-2 3-4 5-6 7 

□□□□□□ 

N/A O 1-2 3-4 5-6 7 

□ □□□□□ 



Notes from school 
m Religious materials 
ft* Legal items (leases, etc.) 

Newspapers/TV Guide 
\ Work materials r 5 




Books/Comics 353. 
Menus/Recipes 
| Magazines 

Other (phone books, bus 
schedules, etc) 

Anything not listed here? 



Easy AlitUe Very 

d o d 

Easy A little Very 

d o d 



Easy A little Very 

d o d 

Easy A little Very 

d o d 



Easy A little Very 

d o d 

Easy A little Very 

d o d 



Easy A little Very 

d o d 

Easy A little Very 

d o d 

Easy A little Very 

d o d 

Easy A little Very 

d o d 



N/A O 1-2 3-4 5-6 7 Easy AlitUe Very 

□ □□□□□ d o d 



Did this list remind you of anything you would like to read during our sessions together? 




Person Reading: Tutor Student (Please check one.) 



Student-Tutor Code # : Date: 

6 — 













Living Literacy: continued from page 2 



dimension in their movement toward enhanced literacy. As beliefs 
are articulated and sometimes restructured through interactions 
with teachers, texts and other learners, the other dimensions of 
development - adults’ practices, processes, goals and plans - 
begin to reflect and then in turn to inform these changes. Although 
these developmental processes appear to be reciprocal and 
recursive, there is evidence that beliefs may be a primary source 
or anchor for other dimensions of growth. 

Understanding adults’ belief systems also requires 
paying attention, initially and over time, to where these 
beliefs have come from, as adults recount and reinterpret 
their past experiences with language and learning by 
describing attitudes, expectations and self-attributions 
acquired during previous schooling. One male literacy 
student, for example, accounted for his own school failure 
by saying: “It’s just like I had told you when we first got 
started that I didn’t learn in school because I was bad - but 
it’s not because I was dumb.” Other adults remember the 
routines of traditional lessons as well as experiences of 
punishment and isolation. 

Documenting changes in beliefs poses at least two 
important problems which can only be mentioned here but merit 
further examination. One is the difficulty many adults experience 
in articulating their underlying assumptions about reading, 
writing, teaching and learning and the second is the appropriate 
role of teachers and teacher-researchers in fostering and 
interpreting these changes. So far we have observed changes in 
the accuracy and complexity of adults’ assessments of their own 
abilities as readers, writers and learners, as well as changes in 
learners’ knowledge of the forms and functions of different texts 
and of what’s entailed in reading and writing for different 
audiences and purposes. Furthermore, adults sometimes begin 
to see their own abilities as less fixed and more malleable; they 
reassess the factors that contribute to and constrain their own 
and others’ (including their children’s) learning, and question 
their own assumptions about the relative merits of one-to-one 
teaching and learning as opposed to participating in a class or 
community of learners. Changes in beliefs may include a more 
critical perspective on literacy and on its cultural uses and effects, 
and these changes may in turn be reflected in revised plans and 
in adults’ processes and patterns of literacy practices in 
everyday life. 

Literacy Practices 

Our second dimension of adult development is literacy 
practices, the range and variation of literacy activities which 
learners engage in their everyday lives. We include here the 
contexts in which adults, individually or collaboratively, use some 
form of written language: the types and uses, settings, and modes 
of engagement. By looking for literacy outside of formal 
educational contexts, researchers found a variety of settings in 
which literacy is acquired spontaneously by individuals in 
response to newly perceived needs in their lives. Changes in the 
practices of adults who are attending programs thus do not 
necessarily reflect program-based learning, nor should we assume 
q individuals who attend literacy programs live in static 




A recurring theme in the literature and in adults’ 
perceptions of practices is the distinction made between literacy 
in and out of “school.” At issue here are the school histories and 
expectations of adult learners, the current school experiences of 
their children, and the extent to which adult educational programs 
resemble school. Defining family literacy as “performing school- 
like literacy activities within the family setting,” according to 
Auerbach (1989), makes the realities of family life an obstacle to 



“Documenting adults’ reading and 
writing through protocols and 
interviews makes literacy processes 
visible and accessible for discussion 
and analysis” 



learning. When family literacy is defined as including a range of 
“activities and practices that are integrated into the fabric of daily 
life, the social context becomes a rich resource that can inform 
rather than impede learning.” Critiquing the assumptions behind 
the “transmission of school practices model,” Auerbach shows 
the complexity of the ways literacy practices are distributed and 
shared in immigrant homes where children and their parents 
engage in the reciprocal or “two-way” support systems and 
stresses the importance of parents becoming their children’s 
advocates at school. 

Documenting changes in literacy practices with adult 
learners over time is obviously a complex process. We have seen 
that adults incorporate new types and uses of literacy into their 
daily lives, or do what they have already been doing more 
frequently or with different modes of engagement. What was 
done individually sometimes becomes a collaborative process, 
while previously supported reading and writing becomes, by 
choice, an activity accomplished independently. Some shifts may 
be only indirectly attributable to new uses of written language, 
but may reflect changes in life patterns which are significant 
byproducts of literacy learning. For example, adults may take 
new roles in their own children’s learning - becoming advocates 
or participants at school; they may become members of 
community, consumer, social or political action groups, or take 
advantage of new resources or support systems. Some changes 
may be individual, others collective. Forms of action may include 
not only individual changes in accomplishing literacy -related 
tasks, but also may signal new roles in class, in the literacy 
program, and in the immediate or broader community (Auerbach 
1990). Finally, tracing changes in practices also involves tracking 
themes or patterns related to content: the subjects that adults in 
particular settings find meaningful to read and write about, their 
substantive analyses and interpretations of important issues. 

Reading and Writing Processes 

In the third dimension of development we include more 
specific information on how adults use oral and written language 
processes, i.e., adult learners’ repertoires of ways to manage 
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particular reading and tasks, and the products of these 
transactions. 

The processes dimension of literacy development 
highlights readers’ and writers’ behaviors immediately before, 
during and after reading and writing, and how these behaviors 
reflect adults’ beliefs. Docu- 
menting adults’ reading and 
writing through protocols and 
interviews makes literacy 
processes visible and accessible 
for discussion and analysis not 
only by teachers and tutors but 
also by the learners themselves. 

Changes over time in processes of reading and writing may be 
observed in adults’ more confident approaches to a range of 
reading and writing tasks, more efficient and effective use of a 
wider range of specific moves and strategies, and more intentional 
selections of processes appropriate for specific purposes. Adult 
readers and writers become increasingly able to deal with more 
complex tasks and aware of alternatives, i.e., different ways to 
accomplish their goals. Some of these processes are observable 
even in group settings as adults collaboratively construct the 
meanings of texts. In addition to, and as evidence of, changes in 
reading and writing processes, the texts adults produce yield 
information about literacy practices and adults’ knowledge and 
perspectives on the issues, events and problems in their 
everyday lives. 

Learners’ Plans 

The fourth dimension, plans, attempts to capture what 
adults themselves indicate they want to learn - their overall 
intentions as well as more specific goals - and the ways they go 
about making learning happen within the instructional context. 
We know that adult learners’ goals in entering literacy programs 
are often not congruent with what funders or program 
designers envision. Planning and setting goals are sometimes 
formalized when adults enter literacy programs, though in 
most cases the processes of negotiation and decision-making 
are ongoing. These processes are a function of complicated 
relationships among adult learners and their own learning, 
their teachers, tutors and co-participants in classes, and their 
co-workers, family members and other community members. 

When adults who enter programs are given the opportunity 
at the outset to explore a range of possibilities, they typically go 
beyond a general interest in “becoming better readers” to name 
particular reading and writing tasks they hope to accomplish, 
often for specific purposes and audiences. Some come with a 
desire to learn more about a particular subject, e.g., African- 
American history, parenting or health. Many seek ways to deal 
with their own children’s literacy and schooling, while others 
wish to participate or assume new roles and responsibilities in 
their families, workplaces or communities. Some are looking 
for community in the literacy program itself. Some seek 
economic improvements in their lives through new jobs or 
promotions, or by dealing more competently with personal 
finances and/or in their encounters with “the bureaucracy.” For 
many the program offers the possibility of taking more control 
or ownership of their own learning. For most adult learners who 
ome to programs, the desire for enhanced self-esteem is implicit 



in many of their stated and unstated goals. 

Central to these planning and goal-setting processes are 
the perceived and enacted roles of tutor or teacher and adult 
learner. Decisions about what to do, how to use time, and who 
should do what (e.g., the roles and expectations of teachers and 

learners) are made and remade 
as instruction evolves. 
Learners may be taught with 
a pre-determined curriculum, 
the curriculum may be 
designed individually to 
address their stated learning 
goals, or teachers and tutors 
may invite learners to explore options and make informed choices 
about alternatives they may not have previously considered. Over 
time, adult learners’ plans and goals appear to change in several 
ways. For adult learners (and perhaps for children and youth as 
well), choice and control may be critical to learning. 

Documenting changes in their plans include identifying 
new goals, becoming more aware of what is involved in 
accomplishing these goals, and determining new priorities. 
Adults’ goals may change from general to more specific short 
or long-term goals. Adults may become increasingly aware of 
constraints or obstacles to change, in themselves and in their 
situation, and they may develop strategies for dealing with these 
constraints or obstacles. Other changes may include more self- 
directedness, enhanced belief in self-efficacy, willingness to take 
risks, increased feelings of safety or comfort as a learner, and a 
propensity to assert plans or goals without explicit prompting. 
Changes in goals may also reflect changes in other dimensions, 
such as in knowledge of the functions and forms of written 
language acquired through experience and interaction with others, 
in and out of literacy programs. For example, in an initial 
interview an adult may indicate that she can already read the 



“For adult learners, choice and 
control may be critical to learning. ” 



newspaper. Later she may name reading the newspaper as a goal, 
having become more acquainted with the parts of the genre or 
with new ways to interpret written information, and/or wishing 
to participate differently in social situations where newspaper 
information is exchanged and discussed. 

Conclusion 

Alternative forms of assessment support teachers and 
learners in working in a more co- investigative fashion, embedding 
assessment in the daily experiences of teaching and learning. 
Literacy portfolios, for example, provide a structure for 
documenting ongoing activities and thus for tracking changes in 
literacy learning over time. These processes help practitioners 
and learners themselves to construct new images of the “living 
literacy” of adults - images built on assumptions of dignity and 
competence, of literacy as reflective and self-critical practice, 
and of learning as participatory. They require that we make 
problematic not only our notions of what counts as development 



“For most adult learners who come 
to programs, the desire for enhanced 
self-esteem is implicit in many of 
their stated and unstated goals.” 



but also our methods of inquiry - the ways we compose, elaborate 
and critique our own frameworks and assessment practices. 

Understanding literacy development in adulthood thus 
requires a rethinking of current assumptions about the needs and 
capabilities of so-called low literate adults, about underlying 
conceptions of literacy, and about the nature of learning in adult 
life. Working together, we need to construct richer conceptual 
frameworks that encompass adults’ beliefs, practices, language 
processes and plans. This process calls for new relationships 
between adult learners and their own knowledge and literacy. 
Many people who work directly with learners in the field are 
concerned about a ceiling on expectations for these adult learners 
who come late to literacy, and about the possibility that pro- 



grams, even the most enlightened, serve to reproduce inequities 
in much the same way as the lowest tracks in high schools become 
permanent repositories for students who have not made it in the 
system. There may be a great deal to be gained if the processes 
of knowledge generation can be shared, and if our methods 
capture and value more of the complexities of human experience. 

(This article is based on research conducted as part of The Adult Literacy 
Evaluation Project at the Literacy Research Center, University of 
Pennsylvania , and supported by the Philadelphia National Bank , The 
Mayor's Commission on Literacy , the UPS Foundation, and the 
Graduate School of Education.) 
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Portfolio Assessment Software 

Portfolio based assessment provides a rich, formative 
picture of student achievement and reveals not only what a student 
has learned, but how he or she is learning. By collecting and 
evaluating actual student work, a teacher can create a clear 
authentic picture of an individual’s strengths and weaknesses. 
Portfolios can be used to document assessment with specialists, 
counselors, administrators, parents and students. 

However, while the advantages of portfolios and authentic 
assessment are clear, organizing and managing the wealth of 
information collected for each student can become an 
overwhelming task. Now, portfolio assessment software, like 
the Grady Profile, makes it possible to maintain a portfolio 
efficiently by allowing educators to collect and organize actual 
exhibits of student work: text (comments, journal entries, 
anecdotes, reflections), sound (reading, talking, singing), graphics 



(writing, math, art), and video (solo or group performances). 
Students and family members may enter evaluations as well as 
the teacher and there are subject cards and over 100 pages 
available for learner objectives. Password-protected security 
limits what authorized users may see and change in the program. 

For further information about portfolio assessment 
software, contact Chip Carlin, LVA-New York State (607-273- 
0634; Internet: Chiplvany@aol.com). 

For information on Grady Profile software contact Aurbach 
& Associates, Inc., 9378 Olive Street Road, Suite 102, St. Louis, 
MO 63132-3222; phone 314-432-7577. 

(Thanks to Aurbach & Associates for permission to reprint the 
above information on portfolio software.) 
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Assessing Workplace Literacy: 

Asking New Questions 

Carol D. Young, Ph.D. 

Consultant to Project REACH 
New York State Governor’s Office of Employee Relations 

Project REACH is a basic skills enhancement program for New York State employees jointly sponsored by the Civil Service 
Employees Association, Inc. and the Governor's Office of Employee Relations. REACH received a grant from the National Institute 
for Literacy to research alternative ways to assess workplace literacy needs. Six researchers observed and interviewed over seventy- 
five workers at six sites. The following article highlights some of their findings and the new questions their research raises about 
assessment. 
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For the National Institute for Literacy Study, we conducted 
observations and interviews with front-line employees at six 
worksites: cleaners and custodians at three sites, nurses’ assistants 
in a metropolitan hospital, developmental aides in a center for 
the developmentally disabled, and clerks in a government office. 
This research was absolutely fascinating. Each site was unique 
with its own culture and communication patterns. Our goal was 
to assess employees’ basic skills needs within the context of their 
work. In our search for alternative assessment approaches, we 
arrived at several important principles but no easy answers. Our 
final report is entitled “Asking New Questions.’’ This article 
highlights a few of our findings and then poses the new questions. 

First, we found that the front-line workers we observed 
were all able to perform their current jobs well. In fact, we 
observed more people who were over-qualified for their positions 
than under-qualified. Their sources of motivation were more 
powerful factors in their performance than their basic skills levels. 
For many custodians, proprietary pride in “my building’’ or “my 
floor” was a source of motivation. It was very frustrating for 
them to be moved to another space without explanation or to be 
prevented from doing a thorough job. For nursing assistants and 
developmental aides, helping another person in need was highly 
rewarding and motivating. The clerks were the one group in the 
study whose motivation often came from outside the work itself. 
With the support of their supervisors, they had elaborate 
“sunshine clubs” to celebrate birthdays, showers, etc. This helped 
break up the monotony of their unrelenting paperwork. 

The clerks were the only worksite in which paperwork, 
reading and writing were central to the job. In all the other sites 
these requirements were more or less peripheral to the main job. 
In the most extreme example we found a “world without words.” 
The researcher observing railroad car cleaners on the night shift 
concluded: “What really goes on is difficult to capture on paper 
- the hours of walking through the trains picking up scraps of 
paper, cups, candy/gum/chips wrappers, etc. or the hours of 
scrubbing and mopping. There isn’t even much dialogue that 
goes on during the work - the work is done very independently. 
Personal conversations go on during little breaks, if at all. In 
short, I feel like I observed a tremendous amount of activity that 
doesn’t amount to much in words.” Reading, writing, speaking 
were largely irrelevant to completing this work. 

Two forces did create a need for higher skill levels: 1) the 
desire to pass civil service exams for advancement, and 2) rapidly 
tanging job requirements in some State agencies. 



In State jobs, civil service exams are a very powerful 
assessment system. For example, Grade 6 clerks have to pass a 
highly competitive written exam to get their job while Grade 5 
and 7 cleaners and janitors do not. However, to advance beyond 
these entry-level positions, they all have to pass fairly rigorous 
exams. Literacy and test-taking skills were important barriers - 
but not the only barriers - to advancement. Some custodians did 
not want to advance to “desk jobs.” They preferred hands-on 
work. In other cases, cutbacks and other organizational factors 
had reduced advancement opportunities severely. 

The accelerating rates of change in some New York State 
agencies were creating some new skills needs as well. For 
example, the Office of Mental Retardation and Developmental 
Disabilities is closing down its large developmental centers and 
moving clients into smaller community residences. This change 
places greater individual responsibility on the developmental 
aides who must function in teams to meet the clients’ needs. 
Also, new record-keeping requirements will require more 
sophisticated writing skills. 

In the sites engaging in “Quality Through Participation,” 
we noted the need for higher level skills in order to participate 
effectively in teams. These skills included having the skills and 
confidence to speak up at meetings as well as knowing how to 
handle conflict. Needless to say, these skill needs extended 
beyond the front-line workers to their supervisors and managers 
as well. 

When we began this research project, we had planned to 
produce a definitive handbook on alternative assessment 
techniques for workplace literacy. We wanted a way to understand 
the organizational context so we could assess employees’ 
capabilities in context. As we researched the six organizations 
and refined our thinking, we came to believe that organizational 
assessment for workplace literacy should be part of an on-going 
collaborative process. It is a process of helping all the stakeholders 
in an organization ask critical questions about skills upgrading 
and organizational barriers and incentives to learning. Ideally, 
the assessment builds a foundation for long-range planning. 

Our report concludes with important questions a workplace 
literacy assessment needs to answer: 

• What is the social and organizational context for using 
basic skills? 

• How do workers perceive their needs for basic skills? 
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• What are organizational and social barriers and incentives 
to learning? 

• If the workplace is trying to solve a problem, is it a 
literacy problem? 

• Who are the stakeholders? How will they agree on defining 
and addressing the problem? 

• What are the short and long-term solutions? How will we 
know we’re progressing? 

• Is individual assessment needed? At what point? 

Our most important conclusion about assessment is that 
organizational literacy programs require organizational 
assessments. If a workplace is trying to address workplace 
problems in a literacy program, the first most critical step is to 
assess the organization’s needs — not the individual employees. 
This organizational assessment requires an active partnership 



among the stakeholders: the union, management, front-line 
supervisors, potential program participants, training officers, 
educational providers, and others as appropriate. 

Only after the organization is assessed does it make sense 
to move onto what we usually think of as assessment, that is, 
assessment of the individual. To assess the individual first implies 
that the individual’s skills are the cause of the problem. Our 
findings support management theorists such as Deming who state 
that usually only 15% of organizational problems are related to 
employee skills and 85% are caused by the system. 

As educators we need to be careful about promising to fix 
problems that have nothing to do with skills, and we want to 
guard against blaming employees for problems they don’t control. 
In educating people for business results this useful formula 
summarizes an important principle (from the book Training for 
Impact , Robinson and Robinson, 1989): 



Education 
or Training 


X 


The Business 
Environment 


= 


The Business 
Result 



In other words, learning that is reinforced on the job will 
contribute to business results. But by the same formula, any 
education or training times zero support back on the job will 



result in zero business results. Improving people’s skill levels 
may create personal results but won’t affect the business unless 
the business supports those new skills. 



Copies of the full report Assessing Workplace Literacy: Asking New Questions is available from Ira Baumgarten, CSEA/LEAP, 
One Lear Jet Lane, Latham, NY 121 10, phone 518-785-4669. 



□ From the Editor’s Desk 



□ 



LVA-New York State’s Director of Education, Ginny 
Gilbertson, has compiled an annotated bibliography on alternative 
assessment for adult literacy. Most of these readings have 
sections on assessment of perceived literacy gains and 
transference of literacy skills to real life tasks and/or assessment 
of perceived affective gains. 

The annotated bibliography is posted to the America Online 
Adult Literacy & Education Forum (in the Programs and Training 
area — Assessment Folder). If you’re not currently linked to 
America Online and would like free software, call 800-827-6364. 
There is a monthly minimum charge for connecting to the 
network. For additional information about America Online and 
the Adult Literacy Forum contact Chip Carlin at 607-273-0634. 
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An Assessment Overview 

Kate Toms 
New York State 
Literacy Resource Center 




Education and training programs use assessment 
information to make decisions about program components, to 
assess student development and the attainment of student goals, 
and to measure success of the program as a whole. Like the 
other writers in this publication, I will focus on the use of 
assessment information to make decisions about student 
development. 

A great deal has been written lately about the need to 
change how we assess in education and training programs. 

This new assessment needs to be informative, believable, 
and fair. However, while many of us have no problem 
with the new measurement practices being fair, few can 
accept that all present users of test information will find 
the results of new forms of assessment either believable or 
informative. The issue of believability is crucial to the 
future of these new assessments. 

Information about individual student development is 
used to inform decisions which fall into two broad 
categories: progress within a program and completion of a 
program and transition to something else. Assessing student 
development through a program or curriculum, and 
assessing student development in order to make decisions 
about movement out of a particular curriculum and into 
something else each require appropriate measures and 
instruments. Portfolio assessment, for example, is a measure of 
student development within an area of study. This powerful 
assessment tool is appropriate for informing decisions about 
movement through a program, but is inappropriate to measure 
program completion or certification of program goals. 

Although “learning gains” — based on the use of 
standardized test scores — are widely accepted by policy makers 
and often accepted by program managers, they are also a point 
of contention within the education community. The argument 
centers around whether the questions on these tests and the form 
of the questions actually measure learning. Inappropriate claims 
for the data provided by standardized testing along with “over 
interpretation” of the test results have led to a questioning of 
their use in general. 

For “alternative assessments,” the major hurdle is that their 
indications of student development are not as easily understood 
or believable as the information gleaned from more traditional 
assessment. While many educators argue that numeric data is 
objective and therefore better, this is not true. Choices about 
what to ask, how to word the stem of the question, which 
alternatives to include as answer choices, the number of items 
on a given curricular domain, etc. render all testing subjective. 



If I ask you what time it is I may never know that you are aware 
that it is snowing. Proponents of alternative modes of assessment 
argue that they give students the chance to indicate what they 
know and how well they know it. Judgments about the quality 
of the portfolio are not any more subjective than the judgments 
of learning based on standardized test scores. For example, one 
teacher can ask that another teacher select a random sample of 
portfolios (which have already been evaluated) and make an 



“Proponents of alternative modes of 
assessment argue that they give 
students the chance to indicate 
what they know and how well they 
know it. Judgments about the 
quality of the portfolio are not any 
more subjective than the judgments 
of learning based on standardized 
test scores. ” 



assessment of the quality of the contents and the validity of the 
first teacher’s decisions about student development. If the two 
expert judges are in agreement about a random sample of 
portfolios, then it can be assumed that they will be in agreement 
about all of the portfolios in the group. The establishment of 
this “inter-rater reliability” has an established methodology in 
research. 

The United States bought into standardized measurement 
as a sole source of information in education and training programs 
to an extent unknown in other countries. The good news is that 
the use of alternative assessments has a long history in other 
countries, where the information gathered through their use is 
considered both believable and fair. The validity and reliability 
of these measures, through the application of sociological 
research techniques, has been established in those 
environments — allowing us to use other forms of assessment 
with a clear understanding of the strengths and weaknesses 
inherent in them. There is still no magic assessment tool which 
will provide all things to all people, but the ongoing, open 
discussion and exchange of our educational experiences and ideas 
will only improve the assessment choices in the future. 



For further information, contact Kate Toms, NYS Literacy Resource Center, State University of New York, 135 Washington Ave., 
O' om 208 Husted, Albany, NY 1 2222. 5 1 8-442-3762. 
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PortfothrAssessment 

Barbara Van Horn, 

Institute for the Study of Adult Literacy 



Traditionally, changes in literacy skills have been measured 
through the use of standardized, norm-referenced tests. While 
these tests are of some value in comparing state or national data, 
they are of limited value to the teacher or to the learner in an 
adult literacy program. In the past few years, interest has grown 
in the use of alternative assessments, such as criterion-referenced, 
competency-based, and curriculum-based assessments which 
more clearly assess the learner’s progress against specific 
indicators or against the curriculum. 

Portfolio assessment is an alternative performance 
assessment that has promise as an effective approach to tracking 
changes in literacy skills for adult learners. Portfolio assessment 
provides a system for evaluating the results of various alternative 
assessments (such as observations, self-ratings, writing samples, 
and cloze tests) as well as the results of norm-referenced tests to 
make instructional decisions and to track progress toward both 
individual and programmatic goals. Key to the successful use 
of portfolio assessments with adults is the active involvement of 
the adult learner through each step of the process — from 
determining portfolio contents through deciding procedures for 
evaluating these collections. 

Portfolios may be particularly effective for several 
reasons: 

• Portfolio assessment requires active and ongoing 
participation by the learner in evaluating the contents of his/ 
her portfolio. Adult learning theory indicates that active 
involvement in the learning process is essential in developing 
self-esteem and skills necessary for self-directed learning. 



(Reprinted with permission - Mosaic: Research Notes on Literacy. 
June 1993, Vol. 3, No. 2. Institute for The Study of Adult Literacy.) 



• Successful portfolio assessment depends on the ongoing 
communication between the instructor and the learner in 
evaluating the portfolio contents. This communication also 
fosters a climate of mutual inquiry and encourages 
participants to take responsibility for their own learning. 

• Portfolio assessments are useful in illustrating relatively 
minor changes in literacy skills. This is particularly important 
in working with adults reading at the lowest levels and those 
with limited English proficiency. 

Portfolios can be an effective way to assess adult learning; 
however, instructors should be aware of several drawbacks to 
using portfolios. This type of assessment does not replace the 
use of norm-referenced tests since the portfolio contents are 
unique to the individual. Generally, portfolios are used in addition 
to norm-referenced assessments that are required by many 
funding agents. In addition, while instructors need to outline 
benchmarks that can be used to evaluate portfolio contents, it is 
not likely that the instructors will find “canned” benchmark 
examples that will serve their learners’ specific needs or the 
requirements of the instructional program. Instructors will have 
to construct reliable examples that illustrate various levels of 
achievement. Finally, portfolio assessment is time-consuming. 
Instructors must work with learners to determine the portfolio 
contents and how it will be evaluated. They also must meet with 
each learner on a regular basis to review and evaluate the 
portfolio. Program developers considering the use of portfolios, 
therefore, should include sufficient time in their instructional 
plans to accommodate the requirements of this promising 
approach to assessment. 
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